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ABSTRACT 

The Austin Independent School District (AISD) became 
interested in evaluation and testing skills of teachers when district 
and school averages on competency ratings in the area were among the 
lowest since 1979. School districts, as well as teacher preparation 
programs, should devote serious attention to the improvement of these 
teacher skills. In the AISD, three initial activities wera undertaken 
to improve teacher competencies in student evaluation: (1) evaluating 
teacher tests to generate development needs and identify resources 
for staff development; (2) creating test item bank projects for staff 
development in a practical format; and (3) long range staff 
development by gathering testing information and conducting training 
in test construction, test interpretation, test use,, and student test 
anxiety. Staff development in testing and evaluation should be a 
districtwide priority. (CM) 
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COLLABORATlur; BETWEEN TESTING AI;D CONTENT SPECIALISTS 
TO IMPROVE TEACHER-MADE TESTS 



Freda M. Holley 
Austin Independent School District 

We in the Austin Independent School District became interested in 
evaluation and testing skills of teachers when we began compiling district and 
school averages on competency ratings given to teachers on the annual teacher 
evaluation form. When ratings on 46 cocapetencies were examined, these were 
among the lowest rated in 1979: 

Prepares appropriate test and evaluation activities to 
measure student learning. 

Interprets own tests and evaluation activities accurately. 

Documents student progress effectively. 
In 1980, 1981, and 1982, this continued to be the case (Totusek, 1982). 

These competencies had initially been included in Austin Independent 
School District's new professional evaluation system because teachers, parents, 
supervisors, and administrators had given these competencies the highest ratings 
of importance. Therefore, these low ratings seemed worthy of real concern. 

Later national research findings came to our attention which contributed 
to our estimate of the importance of teacher testing competencies. For example, 
Ron Edmunds* description of effective schools noted, ''Pupil progress is monitored 
frequently. The teachers use frequent classroom tests to monitor student progress 
toward instructional objectives" (Edmunds, 1979). A SLudy conducted by the 



Center for the Study of Evaluation (1982) found that large amounts of time, as 
such as 10% at the high school level, went to testing; fully three-quarters of 
this testing tinie was consumed with teacher-^developed tests « Yet, the study 
pointed out that while teachers relied mostly on their owri tests for decision- 
niaking, administrators rarely reviewed or evaluated these tests and tended to 
focus their attention on state or district-mandated tests (Center for the 
Study of Evaluation. 1982). 

All of these findings suggest that school districts as well as 
teacher preparation programs should devote serious attention to the improve- 
ment of teacher skills in this area. In Austin ISD, some initial steps in 
this direction have been taken. This paper outlines three activities under- 
taken. 

Evaluating Teacher Tests 
First, administrators had to look at exams. To initiate this, the 
director of secondary education required junior high principals to collect and 
review samples of their teachers' tests. Principals found this activity re- 
vealing. Teachers became aware of the importance of their exam quality and 
appearance. 

The following year both junior and senior high principals collected 
three exams from each teacher. A study of these tests was contracted to local 
university staff (Measurement and Evaluation Center, University of Texas at 
Austin). They scored these exams using the checklist and criteria shown as 
attachments 1 to 3. This study produced a districtwide summary addressed to 
principals. The summary showed that: 

There was good variety in the types of ite ns on the tests. (See 
Table 1, Attachment 4.) There was net too much reliance on one 
item type. 



• Overall, the tests ware good. (See Table 2, Attachment 5.) How- 
tiver, there was also room for improvement. For example, 30% of 
the directions given were not fully adequate. 

Also, some items at every level of Bloom* s Taxonomy appeared on 
the tests. (See Table 3, Attachment 6.) But the percentage of 
items at the higher level was low. 
The study concluded that staff development would be beneficial. 

A script for a faculty meeting was prepared to assist principals in 
returning the exams to teachers. Each teacher received one exam with a feed- 
back sheet. The script also helped principals to explain the purposes of the 
study and to lead the faculty in generating needs and identifying resources 
for staff development on testing competencies. (See Attachment 7.) 

A number of districtwide staff development sessions on testing have 
occurred. However, an improvement need of the kind indicated by our current 
data can only be remedied by a long range intensive effort. 

Test Item Bank Projects 

A more immediate instrumental way to improve teacher tests is to 
work with them to create item banks. AISD^s social studies and science in- 
structional coordinators, with some support from our Office of Research and 
Evaluation, have been working with their teachers for three years to develop 
item banks. In social studies there are currently banks for American History 
and World History. The World History bank is organized by historical periols 
such as "Prehistory." Each item is currently recorded on a card with base 
information on item form, behavior level (from Bloom Taxonomy) , scholastic 
level, stace curriculum goals and objectives and item source. Eventually 
both item banks can be computerized for efficient word processing. 
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Item banks are actually useful in three ways. First, the teachers 
participating in the development receive a great deal of staff development in 
a practical, acceptable format. Through direct and immediate application 
teacher competency is likely to increase. Second, student evaluation is likely 
to be higher in quality through better tests. And, third, teacher time will be 
saved from test development to test production. 

Long-range Staff Development 

The social studies coordinator initiated a staff development effort 
to improve social studies teachers' testing competencies following the receipt 
of the first year's data showing inadequacies in that area. This involved the 
development of a notebook with testing information such as test company bro~ 
chures on definitions and test construction, training sessions with ORE staff 
on test construction and how to deal with student test anxiety, and sessions 
led by the coordinator on interpreting and using standardized test data. 

Conclusion 

Each jf the efforts described rbove is seen as a possible way to 
impact teache*" competencies for student evaluation. Undoubtedly, they have 
been beneficial. The fact remains, however, that the teacher evaluation data 
continues to show that these competencies are low. My experiences indicate 
to me that we are unlikely to make a significant difference in this area unless 
it becomes a real districtwide priority. Unfortunately, district priorities 
tend to focus on desegregation, reorganization, budgets, bond issues, and other 
non-instructional matters. Nevertheless, the promise of payoff, however limited, 
leaves this an area to which we still try to devote whatever attention we can. 
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Attachment 1 



Tetdwr ID No*. 
Subjwrt 



accaptMi staadanis of tnt coa^xoctioD. 



Fc«il»ek oa TeadurMads Tests 

dood Flic 

Appwnc* fjt^ttmr. q «iwfrirtnn, OMtoMi) [1 t 1 

Pmnt for Mch Mcdoa I 1 I ^ 

Good Far 

CtaW tl [1 



Blii|inii 

[ 1 
Sddam 
[ 1 

[ 1 



Th« dMixabk 



Moltipk Choact 



Analyas <rf Itom 

for -Bfa itmi typt w HitKl bitew. If tb* da nctwHri c could not bt XMttd, it w» 



UauOy 



FRir 



tin Fivw tfast tatzoducM ttw itm (stm) cteady 
statn tb* proUna. 

A sbt^ Um it tntsd in Mch item. 

Atoids OM of nvthw «d/or doubk ntptim ia 
ilam. 

Th« Iwfth of wpoMi opttoos ia each iiwi it 
cofijiitrat. 

5^y^n-H>^i cbM to am cofwct answw » 

Tfl* lespoojB optiona in iiutexttd and in cohnno fomL 

Avoids uji of -Att of th« abai»," "Now of tho abot«." 

Trae^abt 

Avoida uaa of -abwyiT and "vrnT In tha it«na. 
Avoids uia of oiptwea in statini *• itama. 
A iinfia idta is testad in each itani. 
Itenu ai« briBf and to tha point. 

Completion 

Only ooa blank ipaca is induded in ^ item. 

Blank spacea occur it the and of tha sentence, 

Gnmniatical dues to tha conect answer are 
ab'^nt. 

Itema aie iwrded to limit the range of poaaibto 
answets. 



Somat im as 

[ I 
[ 1 

[ 1 

[ 1 

[ 1 

[ 1 

[ 1 

[ 1 

C ] 

C ] 

( 1 

C ] 

( 1 
[ 1 
[ ] 
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GL0SSAB7 OF RATING STAHDABDS 



1« App<*ranc« (l»glblllcy» orsanlzatlont nutnus): 

TMtm thm oimnll quality, of th« sppMxanca of th« t<sc» taklTig Ineo 
aeeome haw uay ic is eo nmd t±Lm cmc, its lagiblUtj, MacnMa» thm 
onnnsixmcloa or Inyoat of tbm MmctLonm of tba tMC, md dispUcmdoa qualley. 

Good • CMt Mats eha appaaranea erlearla llacad aiio^ 

Fair *■ cast siaea aavaral* but not oMt, of tha crltarla abova 

MarglxiaL • taat iKsta fav or nona of tha crltarla abova 

Z. Dlractlona ; 

A. Praaant for aach aaetloA — - acan throu^ tht aatlra taat to Idantlfjr 

aach saparata g^nup of Itaa tjpaa; ehack 
to aaa If dLzactlona ara irrlttan out 

Always ■ avary aaparata gsoup of Itaaa la accoapanlad by Iziatruetlona 
Soaatlnaa ■ dlractloius ara usually praaant for aach sactlon 
Saldos ■ wrlttan dlractlons ara praaant for only ona or two aactlona* 
or ara not praaant at all 

B. Clarity — carafuUy raad tha dlractlons that ara prasmt for aach 

aaccLon of Itaoa on tiim taat; rata tha dlractlons in tarms 
of how daarly thay Instruct tha studant aa to what ha/sha 
la to do in answarlng tha quastlons or raspcnding to Itasa* 

Good ■ dlractlons ara daar and unaabiguoua; studant known what to do 
Fair ■ thara Is soma uncartalnty as to how to answar dia itams 
Marginal • It Is difficult to know what to do to answar tha itaiaa» or 
thara ara conflicting inatructioaa; vagua or aidiiguous 

3m Analysis of Itams r 

Ustially • tha Itens that wara samplad mat tha standard or guidallna tha 

majority of tha tim 
Sooatlaias • tha st^dard or guidallna waa mat lass than half of tha tiaa 

for tha itama samplad 
Baraly » only ona or two of tha itams samplad (or nona) mat tha standard 

4« Covaraga of Bloom's Taxonomy : 

Hany of tha Itams * tha majority of tha itams samplad maasurad this laval 
Soma of tha Items * lass than half of tha itams samplad maasurad this laval 
Very Few of tha Itams « only ona or two or nona of tha items samplad 

maasurad this laval 



NOTE : If you ara unable to rata any particular characteristic of the cast, 
leave the response categories blank* 
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TABLE 1 

ITEM TYPES INCLUDED ON THE TESTS 



ITEM TYPE 


YES 


. NO 


Kxiltiple Choice 


49% 


517. 


True /False 


31% 


69% 


Completion 


45% 


55% 


Matching 


42% 


58% 


Essay 


41% 


59% 



n « 1,978 tests 
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Attachment 5 



TABLE 2 



GLOBAL EVALUATIONS OF THE TEACHER-MADE TESTS 



CRITERION 


GOOD 


FAIR 


M^JIGINAL 


Quality of Test 
Appearance 


69% 


25% 


6% 


Adequacy of test 
Directions 


70% 


19% 


11% 


Performance in Meeting 
Item Writing Standards 


59% 


38% 


3% 



n « 1,978 tests 
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TABLE 3 



COVERAGE OF THE 


LEVELS 


OF BLOOM'S 


TAXONOM? 




MANY 


SOME 


VERY FEW 


LEVEL 


ITEMS 


ITEMS 


ITEMS 


KTTOWLEDGE 


777. 


157. 


87a 


COMPREHENSION 


207. 


377o 


437. 


APPLICATION 


137. 


337. 


547. 


ANALYSIS 


47o 


237o 


737. 


SYNTHESIS 


1% 


107. 


897o 


EVALUATION 


17o 


14% 


857o 



n » 1,978 tests 



Attachment 7 

department.il student evaluation questionnaire 

Department 

High School 

Date 



A- Areas of strength in student evaluation and testing in this de- 
partment : 

1. 
2. 
3. 
4. 
5. 

B, Areas that we can improve in student evaluation and testing in 
this department : 

1. 
2. 
3. 
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